{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src=\"http://developer.download.nvidia.com/compute/machine-learning/frameworks/nvidia_logo.png\" style=\"width: 90px; float: right;\">\n",
    "\n",
    "# HugeCTR Inference"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Overview\n",
    "\n",
    "In HugeCTR version 3.0, we have supported inference and added it to the Python interface. This notebook explains how to make inference with trained HugeCTR models in Python. For more details of the usage of Python API, please refer to [HugeCTR Python Interface](../docs/python_interface.md).\n",
    "\n",
    "## Table of Contents\n",
    "1. [Build the HugeCTR Python Interface](#1)\n",
    "1. [DCN Inference Demo](#2)\n",
    "1. [API Signatures](#3)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id=\"1\"></a>\n",
    "## 1. Access the HugeCTR Python Interface\n",
    "\n",
    "1. Please make sure that you have started the notebook inside the running NGC docker container: `nvcr.io/nvidia/merlin/merlin-training:0.5`.\n",
    "\n",
    "   A dynamic link to the `hugectr.so` library is installed to the system path `/usr/local/hugectr/lib/`. Besides, this system path is added to the environment variable `PYTHONPATH`, which means that you can use the Python interface within the docker container environment. Check the dynamic link with the following command:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "hugectr.so  libgmock_main.a  libgtest_main.a\t    libhuge_ctr_static.a\r\n",
      "libgmock.a  libgtest.a\t     libhuge_ctr_shared.so\r\n"
     ]
    }
   ],
   "source": [
    "!ls /usr/local/hugectr/lib"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "2. Import HugeCTR, in order to train your model and to make inference with Python as shown here:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "import hugectr"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id=\"2\"></a>\n",
    "## 2. DCN Inference Demo\n",
    "\n",
    "### 2.1 Download and Preprocess Data\n",
    "1. Download the Kaggle Criteo dataset using the following command:\n",
    "   ```shell\n",
    "   $ cd ${project_root}/tools\n",
    "   $ wget http://azuremlsampleexperiments.blob.core.windows.net/criteo/day_1.gz\n",
    "   ```\n",
    "\n",
    "   In preprocessing, we will further reduce the amounts of data to speedup the preprocessing, fill missing values, remove the feature values whose occurrences are very rare, etc. Here we choose pandas preprocessing method to make the dataset ready for HugeCTR training.\n",
    "\n",
    "2. Preprocessing by Pandas using the following command:\n",
    "   ```shell\n",
    "   $ bash preprocess.sh 1 dcn_data pandas 1 0\n",
    "   ```\n",
    "   \n",
    "   The first argument represents the dataset postfix. It is 1 here since day_1 is used. The second argument dcn_data is where the preprocessed data is stored. The fourth arguement (one after pandas) 1 embodies that the normalization is applied to dense features. The last argument 0 means that the feature crossing is not applied."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2.2 DCN Model Training\n",
    "\n",
    "We can train fom scratch and store the trained dense model and embedding table in model files by doing the following: \n",
    "\n",
    "1. Create a JSON file for the DCN model. \n",
    "   **NOTE**: Please note that the solver clause no longer needs to be added to the JSON file when using the Python interface. Instead, you can configure the parameters using `hugectr.solver_parser_helper()` directly in the Python interface."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Writing dcn_train.json\n"
     ]
    }
   ],
   "source": [
    "%%writefile dcn_train.json\n",
    "{\n",
    "  \"optimizer\": {\n",
    "    \"type\": \"Adam\",\n",
    "    \"update_type\": \"Global\",\n",
    "    \"adam_hparam\": {\n",
    "      \"learning_rate\": 0.001,\n",
    "      \"beta1\": 0.9,\n",
    "      \"beta2\": 0.999,\n",
    "      \"epsilon\": 0.0000001\n",
    "    }\n",
    "  },\n",
    "  \"layers\": [\n",
    "    {\n",
    "      \"name\": \"data\",\n",
    "      \"type\": \"Data\",\n",
    "      \"source\": \"./dcn_data/file_list.txt\",\n",
    "      \"eval_source\": \"./dcn_data/file_list_test.txt\",\n",
    "      \"check\": \"Sum\",\n",
    "      \"label\": {\n",
    "        \"top\": \"label\",\n",
    "        \"label_dim\": 1\n",
    "      },\n",
    "      \"dense\": {\n",
    "        \"top\": \"dense\",\n",
    "        \"dense_dim\": 13\n",
    "      },\n",
    "      \"sparse\": [\n",
    "        {\n",
    "          \"top\": \"data1\",\n",
    "          \"type\": \"DistributedSlot\",\n",
    "          \"max_feature_num_per_sample\": 30,\n",
    "          \"slot_num\": 26\n",
    "        }\n",
    "      ]\n",
    "    },\n",
    "    {\n",
    "      \"name\": \"sparse_embedding1\",\n",
    "      \"type\": \"DistributedSlotSparseEmbeddingHash\",\n",
    "      \"bottom\": \"data1\",\n",
    "      \"top\": \"sparse_embedding1\",\n",
    "      \"sparse_embedding_hparam\": {\n",
    "        \"max_vocabulary_size_per_gpu\": 1447751,\n",
    "        \"embedding_vec_size\": 16,\n",
    "        \"combiner\": 0\n",
    "      }\n",
    "    },\n",
    "    {\n",
    "      \"name\": \"reshape1\",\n",
    "      \"type\": \"Reshape\",\n",
    "      \"bottom\": \"sparse_embedding1\",\n",
    "      \"top\": \"reshape1\",\n",
    "      \"leading_dim\": 416\n",
    "    },\n",
    "    {\n",
    "      \"name\": \"concat1\",\n",
    "      \"type\": \"Concat\",\n",
    "      \"bottom\": [\n",
    "        \"reshape1\",\n",
    "        \"dense\"\n",
    "      ],\n",
    "      \"top\": \"concat1\"\n",
    "    },\n",
    "    {\n",
    "      \"name\": \"slice1\",\n",
    "      \"type\": \"Slice\",\n",
    "      \"bottom\": \"concat1\",\n",
    "      \"ranges\": [\n",
    "        [\n",
    "          0,\n",
    "          429\n",
    "        ],\n",
    "        [\n",
    "          0,\n",
    "          429\n",
    "        ]\n",
    "      ],\n",
    "      \"top\": [\n",
    "        \"slice11\",\n",
    "        \"slice12\"\n",
    "      ]\n",
    "    },\n",
    "    {\n",
    "      \"name\": \"multicross1\",\n",
    "      \"type\": \"MultiCross\",\n",
    "      \"bottom\": \"slice11\",\n",
    "      \"top\": \"multicross1\",\n",
    "      \"mc_param\": {\n",
    "        \"num_layers\": 6\n",
    "      }\n",
    "    },\n",
    "    {\n",
    "      \"name\": \"fc1\",\n",
    "      \"type\": \"InnerProduct\",\n",
    "      \"bottom\": \"slice12\",\n",
    "      \"top\": \"fc1\",\n",
    "      \"fc_param\": {\n",
    "        \"num_output\": 1024\n",
    "      }\n",
    "    },\n",
    "    {\n",
    "      \"name\": \"relu1\",\n",
    "      \"type\": \"ReLU\",\n",
    "      \"bottom\": \"fc1\",\n",
    "      \"top\": \"relu1\"\n",
    "    },\n",
    "    {\n",
    "      \"name\": \"dropout1\",\n",
    "      \"type\": \"Dropout\",\n",
    "      \"rate\": 0.5,\n",
    "      \"bottom\": \"relu1\",\n",
    "      \"top\": \"dropout1\"\n",
    "    },\n",
    "    {\n",
    "      \"name\": \"fc2\",\n",
    "      \"type\": \"InnerProduct\",\n",
    "      \"bottom\": \"dropout1\",\n",
    "      \"top\": \"fc2\",\n",
    "      \"fc_param\": {\n",
    "        \"num_output\": 1024\n",
    "      }\n",
    "    },\n",
    "    {\n",
    "      \"name\": \"relu2\",\n",
    "      \"type\": \"ReLU\",\n",
    "      \"bottom\": \"fc2\",\n",
    "      \"top\": \"relu2\"\n",
    "    },\n",
    "    {\n",
    "      \"name\": \"dropout2\",\n",
    "      \"type\": \"Dropout\",\n",
    "      \"rate\": 0.5,\n",
    "      \"bottom\": \"relu2\",\n",
    "      \"top\": \"dropout2\"\n",
    "    },\n",
    "    {\n",
    "      \"name\": \"concat2\",\n",
    "      \"type\": \"Concat\",\n",
    "      \"bottom\": [\n",
    "        \"dropout2\",\n",
    "        \"multicross1\"\n",
    "      ],\n",
    "      \"top\": \"concat2\"\n",
    "    },\n",
    "    {\n",
    "      \"name\": \"fc4\",\n",
    "      \"type\": \"InnerProduct\",\n",
    "      \"bottom\": \"concat2\",\n",
    "      \"top\": \"fc4\",\n",
    "      \"fc_param\": {\n",
    "        \"num_output\": 1\n",
    "      }\n",
    "    },\n",
    "    {\n",
    "      \"name\": \"loss\",\n",
    "      \"type\": \"BinaryCrossEntropyLoss\",\n",
    "      \"bottom\": [\n",
    "        \"fc4\",\n",
    "        \"label\"\n",
    "      ],\n",
    "      \"top\": \"loss\"\n",
    "    }\n",
    "  ]\n",
    "}\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "2. Write the Python script for training."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Writing dcn_train.py\n"
     ]
    }
   ],
   "source": [
    "%%writefile dcn_train.py\n",
    "from hugectr import Session, solver_parser_helper\n",
    "import sys\n",
    "from mpi4py import MPI\n",
    "\n",
    "def dcn_train(json_file):\n",
    "  solver_config = solver_parser_helper(seed = 0,\n",
    "                                     batchsize = 2048,\n",
    "                                     batchsize_eval =2048,\n",
    "                                     model_file = \"\",\n",
    "                                     embedding_files = [],\n",
    "                                     vvgpu = [[0]],\n",
    "                                     use_mixed_precision = False,\n",
    "                                     scaler = 1.0,\n",
    "                                     i64_input_key = False,\n",
    "                                     use_algorithm_search = True,\n",
    "                                     use_cuda_graph = True,\n",
    "                                     repeat_dataset = True\n",
    "                                    )\n",
    "  sess = Session(solver_config, json_file)\n",
    "  sess.start_data_reading()\n",
    "  for i in range(10000):\n",
    "    sess.train()\n",
    "    if (i%200 == 0):\n",
    "      loss = sess.get_current_loss()\n",
    "      print(\"[HUGECTR][INFO] iter: {}; loss: {}\".format(i, loss))\n",
    "    if (i%1000 == 0 and i != 0):\n",
    "      sess.check_overflow()\n",
    "      sess.copy_weights_for_evaluation()\n",
    "      data_reader_eval = sess.get_data_reader_eval()\n",
    "      for _ in range(solver_config.max_eval_batches):\n",
    "        sess.eval()\n",
    "      metrics = sess.get_eval_metrics()\n",
    "      print(\"[HUGECTR][INFO] iter: {}, {}\".format(i, metrics))\n",
    "  sess.download_params_to_files(\"./\", i+1)\n",
    "  return\n",
    "\n",
    "if __name__ == \"__main__\":\n",
    "  json_file = sys.argv[1]\n",
    "  dcn_train(json_file)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Writing dcn_train.sh\n"
     ]
    }
   ],
   "source": [
    "%%writefile dcn_train.sh\n",
    "cd ../tools && \\\n",
    "python3 ../notebooks/dcn_train.py ../notebooks/dcn_train.json"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[29d12h39m32s][HUGECTR][INFO]: Global seed is 772789275\n",
      "[29d12h39m33s][HUGECTR][INFO]: Peer-to-peer access cannot be fully enabled.\n",
      "Device 0: GeForce RTX 2080 Ti\n",
      "[29d12h39m33s][HUGECTR][INFO]: cache_eval_data is not specified using default: 0\n",
      "[29d12h39m33s][HUGECTR][INFO]: num_workers is not specified using default: 12\n",
      "[29d12h39m33s][HUGECTR][INFO]: num of DataReader workers: 12\n",
      "[29d12h39m33s][HUGECTR][INFO]: max_nnz is not specified using default: 30\n",
      "[29d12h39m33s][HUGECTR][INFO]: num_internal_buffers 1\n",
      "[29d12h39m33s][HUGECTR][INFO]: num_internal_buffers 1\n",
      "[29d12h39m33s][HUGECTR][INFO]: max_vocabulary_size_per_gpu_=1447751\n",
      "[29d12h39m40s][HUGECTR][INFO]: gpu0 start to init embedding\n",
      "[29d12h39m40s][HUGECTR][INFO]: gpu0 init embedding done\n",
      "[HUGECTR][INFO] iter: 0; loss: 0.8440738916397095\n",
      "[HUGECTR][INFO] iter: 200; loss: 0.14668972790241241\n",
      "[HUGECTR][INFO] iter: 400; loss: 0.14474165439605713\n",
      "[HUGECTR][INFO] iter: 600; loss: 0.1524054855108261\n",
      "[HUGECTR][INFO] iter: 800; loss: 0.11932491511106491\n",
      "[HUGECTR][INFO] iter: 1000; loss: 0.1528439223766327\n",
      "[HUGECTR][INFO] iter: 1000, [('AUC', 0.7457985281944275)]\n",
      "[HUGECTR][INFO] iter: 1200; loss: 0.15286946296691895\n",
      "[HUGECTR][INFO] iter: 1400; loss: 0.11360467225313187\n",
      "[HUGECTR][INFO] iter: 1600; loss: 0.13342246413230896\n",
      "[HUGECTR][INFO] iter: 1800; loss: 0.13910263776779175\n",
      "[HUGECTR][INFO] iter: 2000; loss: 0.12295570224523544\n",
      "[HUGECTR][INFO] iter: 2000, [('AUC', 0.7559559941291809)]\n",
      "[HUGECTR][INFO] iter: 2200; loss: 0.1408357471227646\n",
      "[HUGECTR][INFO] iter: 2400; loss: 0.12967847287654877\n",
      "[HUGECTR][INFO] iter: 2600; loss: 0.11960329860448837\n",
      "[HUGECTR][INFO] iter: 2800; loss: 0.1474674493074417\n",
      "[HUGECTR][INFO] iter: 3000; loss: 0.1205996423959732\n",
      "[HUGECTR][INFO] iter: 3000, [('AUC', 0.7648666501045227)]\n",
      "[HUGECTR][INFO] iter: 3200; loss: 0.1083855926990509\n",
      "[HUGECTR][INFO] iter: 3400; loss: 0.1303543895483017\n",
      "[HUGECTR][INFO] iter: 3600; loss: 0.1521710902452469\n",
      "[HUGECTR][INFO] iter: 3800; loss: 0.14694882929325104\n",
      "[HUGECTR][INFO] iter: 4000; loss: 0.11952861398458481\n",
      "[HUGECTR][INFO] iter: 4000, [('AUC', 0.7620059251785278)]\n",
      "[HUGECTR][INFO] iter: 4200; loss: 0.14177869260311127\n",
      "[HUGECTR][INFO] iter: 4400; loss: 0.14483624696731567\n",
      "[HUGECTR][INFO] iter: 4600; loss: 0.1250566840171814\n",
      "[HUGECTR][INFO] iter: 4800; loss: 0.14534974098205566\n",
      "[HUGECTR][INFO] iter: 5000; loss: 0.09943633526563644\n",
      "[HUGECTR][INFO] iter: 5000, [('AUC', 0.7631568312644958)]\n",
      "[HUGECTR][INFO] iter: 5200; loss: 0.11445347219705582\n",
      "[HUGECTR][INFO] iter: 5400; loss: 0.1083950474858284\n",
      "[HUGECTR][INFO] iter: 5600; loss: 0.1364959329366684\n",
      "[HUGECTR][INFO] iter: 5800; loss: 0.12173867225646973\n",
      "[HUGECTR][INFO] iter: 6000; loss: 0.10568396002054214\n",
      "[HUGECTR][INFO] iter: 6000, [('AUC', 0.7714424729347229)]\n",
      "[HUGECTR][INFO] iter: 6200; loss: 0.12835659086704254\n",
      "[HUGECTR][INFO] iter: 6400; loss: 0.13660567998886108\n",
      "[HUGECTR][INFO] iter: 6600; loss: 0.12785111367702484\n",
      "[HUGECTR][INFO] iter: 6800; loss: 0.11540818959474564\n",
      "[HUGECTR][INFO] iter: 7000; loss: 0.14926756918430328\n",
      "[HUGECTR][INFO] iter: 7000, [('AUC', 0.769340991973877)]\n",
      "[HUGECTR][INFO] iter: 7200; loss: 0.1282593309879303\n",
      "[HUGECTR][INFO] iter: 7400; loss: 0.1346716582775116\n",
      "[HUGECTR][INFO] iter: 7600; loss: 0.10817155987024307\n",
      "[HUGECTR][INFO] iter: 7800; loss: 0.13582296669483185\n",
      "[HUGECTR][INFO] iter: 8000; loss: 0.13714417815208435\n",
      "[HUGECTR][INFO] iter: 8000, [('AUC', 0.7707952857017517)]\n",
      "[HUGECTR][INFO] iter: 8200; loss: 0.11192301660776138\n",
      "[HUGECTR][INFO] iter: 8400; loss: 0.12191269546747208\n",
      "[HUGECTR][INFO] iter: 8600; loss: 0.13816437125205994\n",
      "[HUGECTR][INFO] iter: 8800; loss: 0.1508784294128418\n",
      "[HUGECTR][INFO] iter: 9000; loss: 0.12698622047901154\n",
      "[HUGECTR][INFO] iter: 9000, [('AUC', 0.7741658687591553)]\n",
      "[HUGECTR][INFO] iter: 9200; loss: 0.10388685017824173\n",
      "[HUGECTR][INFO] iter: 9400; loss: 0.12212034314870834\n",
      "[HUGECTR][INFO] iter: 9600; loss: 0.10436642169952393\n",
      "[HUGECTR][INFO] iter: 9800; loss: 0.13829629123210907\n",
      "[29d12h40m34s][HUGECTR][INFO]: Rank0: Dump hash table from GPU0\n",
      "[29d12h40m34s][HUGECTR][INFO]: Rank0: Write hash table <key,value> pairs to file\n",
      "[29d12h40m34s][HUGECTR][INFO]: Done\n"
     ]
    }
   ],
   "source": [
    "!bash dcn_train.sh"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2.3 DCN Model Inference\n",
    "\n",
    "1. Create a JSON file for DCN model inference\n",
    "\n",
    "Check the stored model files that will be used in the inference, and create the JSON file for inference. We should remove the solver and optimizer clauses and add the inference clause in the JSON file. The paths of the stored dense model and sparse model(s) should be specified at \"dense_model_file\" and \"sparse_model_file\" within the inference clause. We need to make some modifications to \"data\" in the layers clause. Besides, we need to change the last layer from BinaryCrossEntropyLoss to Sigmoid. The rest of \"layers\" should be exactly the same as that in the training JSON file."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "../tools/0_sparse_10000.model  ../tools/_dense_10000.model\r\n"
     ]
    }
   ],
   "source": [
    "!ls ../tools/*.model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Writing dcn_inference.json\n"
     ]
    }
   ],
   "source": [
    "%%writefile dcn_inference.json\n",
    "{\n",
    "  \"inference\": {\n",
    "    \"max_batchsize\": 4096,\n",
    "    \"dense_model_file\": \"../tools/_dense_10000.model\",\n",
    "    \"sparse_model_file\": \"../tools/0_sparse_10000.model\"\n",
    "  },\n",
    "  \"layers\": [\n",
    "    {\n",
    "      \"name\": \"data\",\n",
    "      \"type\": \"Data\",\n",
    "      \"check\": \"Sum\",\n",
    "      \"label\": {\n",
    "        \"label_dim\": 1\n",
    "      },\n",
    "      \"dense\": {\n",
    "        \"top\": \"dense\",\n",
    "        \"dense_dim\": 13\n",
    "      },\n",
    "      \"sparse\": [\n",
    "        {\n",
    "          \"top\": \"data1\",\n",
    "          \"type\": \"DistributedSlot\",\n",
    "          \"max_feature_num_per_sample\": 30,\n",
    "          \"slot_num\": 26\n",
    "        }\n",
    "      ]\n",
    "    },\n",
    "    {\n",
    "      \"name\": \"sparse_embedding1\",\n",
    "      \"type\": \"DistributedSlotSparseEmbeddingHash\",\n",
    "      \"bottom\": \"data1\",\n",
    "      \"top\": \"sparse_embedding1\",\n",
    "      \"sparse_embedding_hparam\": {\n",
    "        \"max_vocabulary_size_per_gpu\": 1447751,\n",
    "        \"embedding_vec_size\": 16,\n",
    "        \"combiner\": 0\n",
    "      }\n",
    "    },\n",
    "    {\n",
    "      \"name\": \"reshape1\",\n",
    "      \"type\": \"Reshape\",\n",
    "      \"bottom\": \"sparse_embedding1\",\n",
    "      \"top\": \"reshape1\",\n",
    "      \"leading_dim\": 416\n",
    "    },\n",
    "    {\n",
    "      \"name\": \"concat1\",\n",
    "      \"type\": \"Concat\",\n",
    "      \"bottom\": [\n",
    "        \"reshape1\",\n",
    "        \"dense\"\n",
    "      ],\n",
    "      \"top\": \"concat1\"\n",
    "    },\n",
    "    {\n",
    "      \"name\": \"slice1\",\n",
    "      \"type\": \"Slice\",\n",
    "      \"bottom\": \"concat1\",\n",
    "      \"ranges\": [\n",
    "        [\n",
    "          0,\n",
    "          429\n",
    "        ],\n",
    "        [\n",
    "          0,\n",
    "          429\n",
    "        ]\n",
    "      ],\n",
    "      \"top\": [\n",
    "        \"slice11\",\n",
    "        \"slice12\"\n",
    "      ]\n",
    "    },\n",
    "    {\n",
    "      \"name\": \"multicross1\",\n",
    "      \"type\": \"MultiCross\",\n",
    "      \"bottom\": \"slice11\",\n",
    "      \"top\": \"multicross1\",\n",
    "      \"mc_param\": {\n",
    "        \"num_layers\": 6\n",
    "      }\n",
    "    },\n",
    "    {\n",
    "      \"name\": \"fc1\",\n",
    "      \"type\": \"InnerProduct\",\n",
    "      \"bottom\": \"slice12\",\n",
    "      \"top\": \"fc1\",\n",
    "      \"fc_param\": {\n",
    "        \"num_output\": 1024\n",
    "      }\n",
    "    },\n",
    "    {\n",
    "      \"name\": \"relu1\",\n",
    "      \"type\": \"ReLU\",\n",
    "      \"bottom\": \"fc1\",\n",
    "      \"top\": \"relu1\"\n",
    "    },\n",
    "    {\n",
    "      \"name\": \"dropout1\",\n",
    "      \"type\": \"Dropout\",\n",
    "      \"rate\": 0.5,\n",
    "      \"bottom\": \"relu1\",\n",
    "      \"top\": \"dropout1\"\n",
    "    },\n",
    "    {\n",
    "      \"name\": \"fc2\",\n",
    "      \"type\": \"InnerProduct\",\n",
    "      \"bottom\": \"dropout1\",\n",
    "      \"top\": \"fc2\",\n",
    "      \"fc_param\": {\n",
    "        \"num_output\": 1024\n",
    "      }\n",
    "    },\n",
    "    {\n",
    "      \"name\": \"relu2\",\n",
    "      \"type\": \"ReLU\",\n",
    "      \"bottom\": \"fc2\",\n",
    "      \"top\": \"relu2\"\n",
    "    },\n",
    "    {\n",
    "      \"name\": \"dropout2\",\n",
    "      \"type\": \"Dropout\",\n",
    "      \"rate\": 0.5,\n",
    "      \"bottom\": \"relu2\",\n",
    "      \"top\": \"dropout2\"\n",
    "    },\n",
    "    {\n",
    "      \"name\": \"concat2\",\n",
    "      \"type\": \"Concat\",\n",
    "      \"bottom\": [\n",
    "        \"dropout2\",\n",
    "        \"multicross1\"\n",
    "      ],\n",
    "      \"top\": \"concat2\"\n",
    "    },\n",
    "    {\n",
    "      \"name\": \"fc4\",\n",
    "      \"type\": \"InnerProduct\",\n",
    "      \"bottom\": \"concat2\",\n",
    "      \"top\": \"fc4\",\n",
    "      \"fc_param\": {\n",
    "        \"num_output\": 1\n",
    "      }\n",
    "    },\n",
    "    {\n",
    "      \"name\": \"sigmoid\",\n",
    "      \"type\": \"Sigmoid\",\n",
    "      \"bottom\": \"fc4\",\n",
    "      \"top\": \"sigmoid\"\n",
    "    }\n",
    "  ]\n",
    "}\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "2. Convert the criteo data to inference format\n",
    "\n",
    "The HugeCTR inference is enabled by predict() method of InferenceSession, which requires dense features, embedding columns and row pointers of slots as the input and gives the prediction result as the output. We need to convert the criteo data to inference format first. Please find the details of input format [here](https://github.com/triton-inference-server/hugectr_backend/blob/main/docs/user_guide.md#variant-compressed-sparse-row-input)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "!python3 ../tools/criteo_predict/criteo2predict.py --src_csv_path=../tools/dcn_data/val/test.txt --src_config=../tools/criteo_predict/dcn_data.json --dst_path=./dcn_csr.txt --batch_size=4096"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    " 3. Write the Python script for inference."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Writing dcn_inference.py\n"
     ]
    }
   ],
   "source": [
    "%%writefile dcn_inference.py\n",
    "import sys\n",
    "from mpi4py import MPI\n",
    "from hugectr.inference import CreateParameterServer, CreateEmbeddingCache, InferenceSession\n",
    "\n",
    "def dcn_inference(config_file, model_name, data_path):\n",
    "  # read data from file\n",
    "  data_file = open(data_path)\n",
    "  labels = [int(item) for item in data_file.readline().split(' ')]\n",
    "  dense_features = [float(item) for item in data_file.readline().split(' ')]\n",
    "  embedding_columns = [int(item) for item in data_file.readline().split(' ')]\n",
    "  row_ptrs = [int(item) for item in data_file.readline().split(' ')]\n",
    "  # create parameter server, embedding cache and inference session\n",
    "  parameter_server = CreateParameterServer([config_file], [model_name], False)\n",
    "  embedding_cache = CreateEmbeddingCache(parameter_server, 0, True, 0.2, config_file, model_name, False)\n",
    "  inference_session = InferenceSession(config_file, 0, embedding_cache)\n",
    "  # make prediction and calculate accuracy\n",
    "  output = inference_session.predict(dense_features, embedding_columns, row_ptrs)\n",
    "  accuracy = calculate_accuracy(labels, output)\n",
    "  print(\"[HUGECTR][INFO] prediction number samples: {}, accuracy: {}\".format(len(labels), accuracy))\n",
    "\n",
    "def calculate_accuracy(labels, output):\n",
    "  num_samples = len(labels)\n",
    "  flags = [1 if ((labels[i] == 0 and output[i] <= 0.5) or (labels[i] == 1 and output[i] > 0.5)) else 0 for i in range(num_samples)]\n",
    "  correct_samples = sum(flags)\n",
    "  return float(correct_samples)/float(num_samples)\n",
    "    \n",
    "if __name__ == \"__main__\":\n",
    "  config_file = sys.argv[1]\n",
    "  model_name = sys.argv[2]\n",
    "  data_path = sys.argv[3]\n",
    "  dcn_inference(config_file, model_name, data_path)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[29d12h41m36s][HUGECTR][INFO]: default_emb_vec_value is not specified using default: 0.000000\n",
      "[29d12h41m37s][HUGECTR][INFO]: Global seed is 1326185260\n",
      "[29d12h41m38s][HUGECTR][INFO]: Peer-to-peer access cannot be fully enabled.\n",
      "[29d12h41m38s][HUGECTR][INFO]: algorithm_search is not specified using default: 1\n",
      "[29d12h41m38s][HUGECTR][INFO]: cuda_graph is not specified using default: 1\n",
      "[29d12h41m38s][HUGECTR][INFO]: start create embedding for inference\n",
      "[29d12h41m38s][HUGECTR][INFO]: sparse_input name data1\n",
      "[29d12h41m38s][HUGECTR][INFO]: create embedding for inference success\n",
      "[HUGECTR][INFO] prediction number samples: 4096, accuracy: 0.96435546875\n"
     ]
    }
   ],
   "source": [
    "!python3 dcn_inference.py dcn_inference.json DCN dcn_csr.txt"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id=\"3\"></a>\n",
    "## 3. API Signatures\n",
    "\n",
    "Here is the list of all the API signatures within the HugeCTR Python interface related to the inference feature. As you can see from the above example, we have included `CreateParameterServer`, `CreateEmbeddingCache`, and `InferenceSession`.\n",
    "\n",
    "**CreateParameterServer**\n",
    "```bash\n",
    "CreateParameterServer(...) method of builtins.PyCapsule instance\n",
    "    CreateParameterServer(model_config_path: List[str], model_name: List[str], i64_input_key: bool) -> hugectr.inference.ParameterServerBase\n",
    "```\n",
    "\n",
    "**CreateEmbeddingCache**\n",
    "```bash\n",
    "CreateEmbeddingCache(...) method of builtins.PyCapsule instance\n",
    "    CreateEmbeddingCache(parameter_server: hugectr.inference.ParameterServerBase, cuda_dev_id: int, use_gpu_embedding_cache: bool, cache_size_percentage: float, model_config_path: str, model_name: str, i64_input_key: bool) -> hugectr.inference.EmbeddingCacheInterface\n",
    "```\n",
    "\n",
    "**InferenceSession**\n",
    "```bash\n",
    "class InferenceSession(pybind11_builtins.pybind11_object)\n",
    " |  Method resolution order:\n",
    " |      InferenceSession\n",
    " |      pybind11_builtins.pybind11_object\n",
    " |      builtins.object\n",
    " |\n",
    " |  Methods defined here:\n",
    " |\n",
    " |  __init__(...)\n",
    " |      __init__(self: hugectr.inference.InferenceSession, config_file: str, device_id: int, embedding_cache: hugectr.inference.EmbeddingCacheInterface) -> None\n",
    " |\n",
    " |  predict(...)\n",
    " |      predict(*args, **kwargs)\n",
    " |      Overloaded function.\n",
    " |\n",
    " |      1. predict(self: hugectr.inference.InferenceSession, dense_feature: List[float], embeddingcolumns: List[int], row_ptrs: List[int]) -> List[float]\n",
    " |\n",
    " |      2. predict(self: hugectr.inference.InferenceSession, dense_feature: List[float], embeddingcolumns: List[int], row_ptrs: List[int], i64_input_key: bool) -> List[float]\n",
    "\n",
    "```"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.9"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}
